Handheld computer systems and techniques for character and command recognition related to human movements

ABSTRACT

Systems and methods for human hand gesture recognition through a training mode and a recognition mode are disclosed. In the training mode, a user can move a handheld device with a hand gesture intended to represent a command. Sensors within the handheld device can record raw data, which can be processed to obtain a set of values corresponding to a set of discrete features, which is stored in a database and associated with the intended command. The process is repeated for various hand gestures representing different commands. In the recognition mode, the user can move the handheld device with a hand gesture. A computer system can compare a set of values corresponding to a set of discrete features derived from the hand gesture with the sets of values stored in the database, select a command with the closest match and displays and/or executes the command.

CROSS-REFERENCE TO RELATED APPLICATION

Under 35 U.S.C. 120, this application is a Continuation application andclaims priority to U.S. application Ser. No. 12/832,707, filed Jul. 8,2010, entitled “HANDHELD COMPUTER SYSTEMS AND TECHNIQUES FOR CHARACTERAND COMMAND RECOGNITION RELATED TO HUMAN MOVEMENTS,” which claims thebenefit of U.S. Provisional Application Ser. No. 61/259,288, filed Nov.9, 2009, which is entitled “INTELLIGENT COMPUTER SYSTEM AND TECHNIQUESFOR CHARACTER AND COMMAND RECOGNITION RELATED TO HUMAN MOVEMENTS”, allof which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to a handheld computerized human handgesture recognition system, and particularly to a handheld computersystem that includes a handheld device having Micro Electro-MechanicalSystems (MEMS) based inertial sensors including a gyroscope and anaccelerometer, and a miniature compass.

BACKGROUND OF THE INVENTION

A hand gesture is a form of non-verbal communication in which visiblehand actions allow an individual to communicate messages. Hand gesturerecognition is being increasingly used in computer technology, with thegoal of interpreting human gestures via mathematical algorithms. Handgesture recognition technology enables humans to interface with acomputer and interact naturally without any mechanical devices. Forexample, using the concept of gesture recognition, it is possible topoint a finger at the computer screen so that the cursor will moveaccordingly.

The hand gesture recognition concept is also being increasingly used inthe video game technology, in which a player's hand gestures arecommunicated to a computer system by way of a handheld device. Forexample, Wii, a home video game console made by Nintendo Co. Ltd. uses awireless controller, the Wii Remote, which can be used as a handheldpointing device and can detect movement in three dimensions. The Wiiremote can study changes in acceleration over time to representgestures.

There are many challenges associated with the accuracy and usefulness ofconventional gesture recognition computer systems and software.Conventional systems are not highly accurate because they generally relyon linear sensors, whereas human hand gestures largely include angularmovements. Thus, conventional systems can only properly recognize largehand movements and are unable to properly differentiate between handgestures that are only slightly different from each other. It would bedesirable to have recognition systems that can differentiate betweensimilar hand gestures.

SUMMARY OF THE INVENTION

The following presents a simplified summary of the subject disclosure inorder to provide a basic understanding of some aspects thereof. Thissummary is not an extensive overview of the various embodiments of thesubject disclosure. It is intended to neither identify key or criticalelements nor delineate any scope. Its sole purpose is to present someconcepts in a simplified form as a prelude to the more detaileddescription that is presented later.

The present invention relates to novel and efficient handheld computersystem and methodology for human hand gesture recognition. The computersystem includes a remote handheld device that includes inertial sensors,for example, a gyroscope, an accelerometer, and/or a compass. In thetraining mode, a user makes an intentional hand gesture while holdingthe handheld device to communicate a command or a character. Datarelated to the gesture movement is captured and provided by the inertialsensors. The data is pre-processed, filtered and converted into a set ofvalues corresponding to a set of discrete features. The set of valuesare stored in a database and a command or a character is associated withthe set. The database is populated for various hand gestures. Duringoperation, the same data capture and conversion steps are performed fora user's hand gestures, and the set of values is compared with thedatabase. The command associated with the match or the closest match isexecuted, or the character associated with the match or the closestmatch is displayed on the display of the handheld device.

To the accomplishment of the foregoing and related ends, the one or moreaspects comprise the features hereinafter fully described andparticularly pointed out in the claims. The following description andthe annexed drawings set forth in detail certain illustrative featuresof the one or more aspects. These features are indicative, however, ofbut a few of the various ways in which the principles of various aspectsmay be employed, and this description is intended to include all suchaspects and their equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and advantages of the present invention willbe apparent upon consideration of the following detailed description,taken in conjunction with the accompanying drawings, in which likereference characters refer to like parts throughout, and in which:

FIG. 1 illustrates an exemplary flow diagram for the methodology of thepresent invention;

FIG. 2 illustrates an exemplary computer network in which the presentinvention can be implemented;

FIG. 3 illustrates an exemplary computing environment in which thepresent invention can be implemented;

FIG. 4 illustrates an exemplary display for providing the user with areal time feedback;

FIG. 5 illustrates an exemplary flow diagram for the training modemethodology of the present invention;

FIG. 6 illustrates an exemplary flow diagram for the recognition modemethodology of the present invention;

FIG. 7 illustrates exemplary computer modules that can be used by thegesture recognition engine of the present invention;

FIG. 8 illustrates an exemplary flow diagram for the operating modemethodology of the present invention;

FIG. 9 illustrates an exemplary algorithm for the operating modemethodology of the present invention;

FIG. 10 illustrates a system according to an exemplary embodiment of thepresent invention, in which the inertial sensors are used for powermanagement;

FIG. 11 illustrates an exemplary power management methodology of thepresent invention;

FIG. 12 illustrates an exemplary handheld device in which the powermanagement techniques of the present invention can be implemented;

FIG. 13 illustrates an exemplary methodology of the present invention inwhich the camera application is switched from the sleep mode to theactive mode;

FIG. 14 illustrates an exemplary methodology of the present invention inwhich the handheld device is switched from the active mode to the sleepmode and back to the active mode;

FIG. 15 illustrates an exemplary handheld device of the presentinvention including inertial and non-inertial sensors;

FIG. 16 illustrates an exemplary methodology of the present inventionfor combining data provided by the inertial sensors;

FIG. 17 illustrates an exemplary methodology of the present invention inwhich angular movement is correlated with linear movement;

FIG. 18 illustrates an exemplary methodology of the present invention inwhich a microphone and inertial sensors are used for userauthentication; and

FIG. 19 illustrates an exemplary methodology of the present inventionfor unlocking a handset screen.

DETAILED DESCRIPTION

This present invention discloses system(s) and techniques forrecognizing character(s) or commands intended by human movements, anddisplaying the characters or executing the commands. The claimed subjectmatter is now described with reference to the drawings, wherein likereference numerals are used to refer to like elements throughout. In thefollowing description, for purposes of explanation, numerous specificdetails and examples are set forth in order to provide a thoroughunderstanding of the subject innovation. One of ordinary skill in theart will understand, however, that the subject innovation may bepracticed without these specific details and is not limited by thesespecific details and examples. One of ordinary skill in the art willalso appreciate that well-known structures and devices are shown inblock diagram form in order to facilitate describing the claimed subjectmatter.

As used in this application, the terms “component,” “module,” “system,”or the like can, but need not, refer to a computer-related entity,either hardware, a combination of hardware and software, software, orsoftware in execution. For example, a component might be, but is notlimited to being, a process running on a processor, a processor, anobject, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on acontroller and the controller can be a component. One or more componentscan reside within a process and/or thread of execution and a componentcan be localized on one computer and/or distributed between two or morecomputers.

Furthermore, the claimed subject matter may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedsubject matter. The term “article of manufacture” as used herein isintended to encompass a computer program accessible from anycomputer-readable device, carrier, or media. For example, computerreadable media can include but are not limited to magnetic storagedevices (e.g., hard disk, floppy disk, magnetic strips . . . ), opticaldisks (e.g., compact disk (CD), digital versatile disk (DVD) . . . ),smart cards, and flash memory devices (e.g., card, stick, key drive . .. ). Additionally it should be appreciated that a carrier wave can beemployed to carry computer-readable electronic data such as those usedin transmitting and receiving electronic mail or in accessing a networksuch as the Internet or a local area network (LAN). Of course, thoseskilled in the art will recognize many modifications may be made to thisconfiguration without departing from the scope or spirit of the claimedsubject matter.

Moreover, the word “exemplary” is used herein to mean serving as anexample, instance, or illustration. Any aspect or design describedherein as “exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects or designs. Rather, use of the wordexemplary is intended to present concepts in a concrete fashion. As usedin this application, the term “or” is intended to mean an inclusive “or”rather than an exclusive “or.” Therefore, unless specified otherwise, orclear from context, “X employs A or B” is intended to mean any of thenatural inclusive permutations. That is, if X employs A; X employs B; orX employs both A and B, then “X employs A or B” is satisfied under anyof the foregoing instances. In addition, the articles “a” and “an” asused in this application and the appended claims should generally beconstrued to mean “one or more” unless specified otherwise or clear fromcontext to be directed to a singular form.

As used herein, the terms “infer” and “inference” generally refer to theprocess of reasoning about or inferring states of the system,environment, and/or user from a set of observations as captured viaevents and/or data. Inference can be employed to identify a specificcontext or action, or can generate a probability distribution overstates, for example. The inference can be probabilistic—that is, thecomputation of a probability distribution over states of interest basedon a consideration of data and events. Inference can also refer totechniques employed for composing higher-level events from a set ofevents and/or data. Such inference results in the construction of newevents or actions from a set of observed events and/or stored eventdata, whether or not the events are correlated in close temporalproximity, and whether the events and data come from one or severalevent and data sources.

MEMS-based motion sensors include an accelerometer and a gyroscope. Anaccelerometer can be used to measure linear acceleration. The physicalmechanisms underlying MEMS-based accelerometers include capacitive,piezoresistive, electromagnetic, piezoelectric, ferroelectric, opticaland tunneling. MEMS-based accelerometers can be simple devicesconsisting of a cantilever beam with a predetermined test mass (alsoknown as proof mass seismic mass). Under the influence of externalaccelerations, the mass deflects from its neutral position. Thisdeflection is measured in an analog or digital manner. Commonly, thecapacitance between a set of fixed beams and a set of beams attached tothe proof mass is measured.

Other types of MEMS-based accelerometers can contain a small heater atthe bottom of a very small dome, which heats the air inside the dome tocause it to rise. A thermocouple on the dome determined where the heatedair reaches the dome and the deflection off the center is a measure ofthe acceleration applied to the sensor. MEMS-based accelerometersgenerally operate in-plane, that is, they are designed to be sensitiveonly to a direction of the plane of the die. By integrating two devicesperpendicularly on a single die a two-axis accelerometer can be made. Byadding an additional out-of-plane device, three axes can be measured.Accelerometers with integral electronics offer readout electronics andself-test capability.

A compass is an instrument used for determining direction relative tothe earth's magnetic pole. It consists of a magnetized pointer free toalign itself with the earth's magnetic field. Miniature compasses areusually built out of two or three magnetic field sensors, for exampleHall sensors, that provide data for a microprocessor. The correctheading relative to the compass is calculated using trigonometry. Often,a miniature compass is a discrete component which outputs either adigital or analog signal proportional to its orientation. This signal isinterpreted by a controller or microprocessor. The compass can usehighly calibrated internal electronics to measure the response of thecompass to the earth's magnetic field. Examples of miniature compassesavailable in the marketplace include the HMC1051Z single-axis and theHMC1052 two-axis magneto-resistive sensors sold by HoneywellInternational Inc., the AK8973 3-axis electronic compass sold by AsahiKasei Microdevices Corporation, and the AMI 201 (2-axis) and the AMI 302(3-axis) electronic compass modules sold by Aichi Micro IntelligentCorporation of Japan.

A gyroscope is a device used for measuring or maintaining orientation,based on the principles of conservation of angular momentum. MEMS-basedgyroscopes use vibrating proof masses. Those masses typically vibrate ata high frequency. As the sensor housing rotates in inertial space aCoriolis force is induced on the proof mass. The Coriolis force causes avibration in an orthogonal plane and the amplitude of the orthogonalmotion can be measured. This type of device is also known as a Coriolisvibratory gyro because as the plane of oscillation is rotated, theresponse detected by the transducer results from the Coriolis term inits equations of motion (“Coriolis force”). A vibrating structuregyroscope can be implemented as a tuning fork resonator, a vibratingwheel or a wine glass resonator using MEMS technology.

One of ordinary skill in the art will appreciate that the presentinvention is not limited to MEMS based devices, that the MEMS basedembodiments disclosed herein are exemplary, and that the presentinvention can be implemented with any accelerometer, compass andgyroscope that can be incorporated in a handheld device. One of ordinaryskill in the art will appreciate that other types of inertial sensorsthat can be included in a handheld device, for example, quartz sensors,can also be used in the present invention. Other types of inertialsensors that include mechanical components on the micron or millimeterscale and can be combined with electronic circuitry can also be used inthe present invention.

FIG. 1 illustrates an exemplary flow diagram 100 for the method of thepresent invention. The flow diagram 100 includes two components, atraining mode component 110 and a recognition mode component 130. In thetraining mode 110, the user begins by causing an intentional movement ofthe handheld device in the air (Step 112). The user's intentionalmovement corresponds to a command or a character. In one embodiment, thehandheld device can include MEMS-based gyroscope and accelerometer. Inanother embodiment, the handheld devices can include MEMS-basedgyroscope and accelerometer and a miniature compass.

At step 114, the data related to the trajectory of the gesture made inStep 112 is extracted by using the gyroscope, accelerometer and compassembedded in the handheld device. As human movement is largely angular, atrajectory drawn in the air with the hand can be expressed primarilywith angular movement using a gyroscope. The raw gyroscope data isproportional to angular velocity, so this data can be preprocessed toremove any offset and integrated to provide an estimate of angularmovement. The gyro bias offset, also known as the null voltage, is themeasured voltage when the gyro is not rotating about its sensitive axis.Gyro output voltage measurements above the bias offset indicate rotationin one direction (e.g., clockwise), while voltage measurements below thebias offset indicate rotation in the opposite direction (e.g.,counter-clockwise).

Additional filtering to clean up the signal may remove involuntary handjitter and other unwanted high frequency components. A jitter is a smalland rapid vibration that is inherently present in a human hand. A deadzone or similar mechanism can be used to remove slow, unintentionalmovement. A dead zone in a sensor is the point, period, area or zonewhere a sensor does not work. InvenSense Inc., the assignee of thepresent patent application, has developed proprietary methodologies,software and architectures for calibrating and correcting errors inhandheld devices having MEMS-based gyroscopes and accelerometers and/orminiature compasses. These innovations allow gesture recognition systemsto interpret and differentiate between the various commands andcharacters intended by the user, even when different commands andcharacters are represented by hand gestures that are only slightlydifferent from each other.

Returning to Step 114, the accelerometer data can also be used to helpstabilize gyroscope bias, and to facilitate determining the direction ofgravity. The accelerometer data can also be correlated with thegyroscope data to distinguish between rotation with a strong linearcomponent, or moment arm, signifying intentional movement through theair, and rotational component without such component, signifyingunintentional rotation of the device within the user's hand. A compasscan also be incorporated in the handheld device to stabilize thegyroscope bias in the yaw direction. The result of this preprocessing isa trajectory containing X and Y data points corresponding to yaw andpitch angular movement. Pitch, roll and yaw refer to movements of anobject that are measured as angles. Pitch is up and down like a box lid.Yaw is left and right like a door on hinges, and roll is rotation.

To help isolate the trajectory, in one embodiment, a button on thehandheld device can be used to distinguish between intentional gesturemovement and unintentional movement or wind-up movement. The button canbe held down during the course of gesture, or it can be pressed once atthe beginning of the gesture and once at the end. In another embodiment,the speed of the trajectory can be used to determine start and end of agesture. For example, when the user begins a large, fast movement, agesture is assumed to have begun. When some time passes and the totalamount or speed of movement is reduced to below some threshold value,the gesture is assumed to have ended. In yet another embodiment, anymovement is assumed to be a potential gesture movement, and a movingwindow of data is used to capture gestures.

As users may not be familiar with the experience of using hand movementto draw predetermined trajectories in the air, feedback can be used tohelp the user learn to control the trajectory. In this case, an imagecorresponding to the X and Y path of the trajectory can be displayed,for example, on a computer display, to notify the user of the trajectorythat will be used in the gesture recognition algorithm. One of ordinaryskill in the art would appreciate that it would useful to normalize thetrajectory in space and time, such that it is suitable for storage,querying and comparison in a database structure. To normalize it inspace, the trajectory can be scaled down by an amount or ratiodetermined by the maximum of the total X and Y spans. To normalize intime, averaging can be used to reduce the number of data points. Afternormalization, all trajectories would have the same size and the samenumber of data points, thereby making it easier to compare them byshape.

At Step 116, the trajectory data can now be converted to a set valuescorresponding to a set of selected discrete features. One example of aset of features is the angle of the trajectory at any given point intime. This can be calculated using the inverse tangent of a change inpitch divided by a change in yaw. Another example of a set of featuresis the radial squared component, comprising X squared plus Y squared.Another example of a set of features is the pitch component, comprisingonly the pitch, or Y data points. Another example of a set of featuresis the yaw component, comprising only the yaw, or X data points. Anotherexample of a set of features is the rate of change of the trajectory,useful for distinguishing, for example, the letter “U” from the letter“V.” Another example of a set of features is the location of a crossingpoint, useful for distinguishing, for example, the number “8” from thenumber “0.” Another example of a set of features is the number of timeswithin a trajectory in which the trajectory changes direction, with thelimitation that the angular change in direction is greater than athreshold value. One of ordinary skill in the art will appreciate thatother sets of similar types of features can also be used within thepremise and objectives of the system of the present invention.

At Step 118, after a set of values corresponding to a predetermined setof discrete features has been created, the set of values can be storedin a database along with a corresponding identifier, for example, an IDnumber, indicating the desired gesture. At Step 120, the database can bepopulated with numerous sets of values corresponding to the desiredfeatures and associated with various identifiers and desired gestures.The following is an exemplary database that can be created in Step 120.The table below includes a set of N features and M sets of values forthe N features. A command or a character is associated with each set ofvalues.

Feature Feature Feature Feature Command/ 1 2 3 . . . N Character Set 1Set 2 Set 3 . . . Set M

The database populated during the training mode 110 can then be used forgesture recognition mode during the recognition mode 130. In therecognition mode, at Step 132, the user makes a hand gesture either bymoving the hand held device or just his/her hand in the air. At Step134, the handheld device extracts data related to the trajectory of thegesture made in Step 132 by using the gyroscope, accelerometer andcompass embedded in the handheld device.

At Step 136, the trajectory data extracted in Step 134 is converted intoa set of values corresponding to some discrete features. Steps 134 and136 can be implemented in the same or similar manner that Steps 114 and116 in the training mode 110 are implemented respectively. At Step 138,the set of values that has been determined from the input trajectory inStep 136 can be compared to all sets of values stored in a gesturelibrary or a subset of the sets of values stored in the gesture library.This library was generated by using training data in Step 120, or byusing hard-coded data. At Step 140, upon identifying the set of valuesfrom the library that matches or most closely matches the set of valuesgenerated by the user's input trajectory, the character corresponding tothat set of values is displayed or the command corresponding to that setof values is executed. The character can include a letter from analphabet, which can be English or another language. One of ordinaryskill in the art will appreciate that the term “character” as used canalso include a word, a phrase or a signature. The character can alsoinclude a number, for example, a Roman numeral. The character can bedisplayed can be displayed in a word processing application document,for example, Microsoft Word.

At Step 140, to compare user input values set corresponding to a featureset with values sets corresponding to the feature set within thelibrary, various techniques exist that can be used, including look-uptables, Hidden Markov Models (HMM), neural networks, and support vectormachines. In the optimal embodiment, HMMs are used with somemodification for optimization. An HMM includes a state transition matrixand an emission matrix. In the state transition matrix, a left-rightmodel is used in which each state must be followed by the next state:

-   -   0 1 0 0 0 0 etc.    -   0 0 1 0 0 0 etc.    -   0 0 0 1 0 0 etc.    -   0 0 0 0 1 0 etc.    -   0 0 0 0 0 1 etc.    -   etc.

In the emission matrix, the features from the library can be used togenerate a matrix in which the optimal point in the matrix is given byeach library value for a feature, but with some allowed probability oneither side of the library value for a feature. For example, if a givenstate has been recorded with feature #4 for a given library feature set,the corresponding row in the emission matrix might include: 0 A B C B A0 0 etc

In which C corresponds to a high probability, and B and A correspond torespectively lower probabilities. This matrix may be generated usingtraining methods such as the Baum-Welch method or may be entereddirectly from the feature library. After the state transition matrix andemission matrix have been generated, they may be used to evaluate theprobability that input feature set values matches the library featuresset values using standard algorithms such as the forward algorithm orthe Viterbi algorithm. The final probability may be generated as a login order to prevent excessive scaling differences.

When several different feature sets are used to evaluate an inputtrajectory, a probability will be generated for each feature set. Forexample, probability A may reflect the probability for angular features,probability B may reflect the probability for radial features, and soon. A final probability can be given as a weighted linear combination ofall the probabilities for all the types of features, in which theweights are determined experimentally. The highest probabilitydetermines best match for the trajectory; if the highest probability isbelow a threshold, then no match is returned.

In one embodiment, a set of short gestures may be pre-trained, such as aset of numbers, characters, or other trajectories of similar lengths.For each gesture, a set of different trajectories may be trained; forexample, a 2 may be trained with or without a loop in the lower leftcorner. When a user inputs a trajectory that matches one of thepredetermined feature sets, an event is triggered in the application oroperating system of the device. In one embodiment, the gesture maytrigger a mode change or event within an application or game. In anotherembodiment, the gesture may open or close an application, or otherwisecontrol the set of applications running within an operating system. Inanother embodiment, a number or letter may indicate an element within alist within a program or operating system.

In one example, in a contact list within a phone, drawing a letter mayindicate that the contact list should scroll to that letter within thealphabet. In another example, within a main menu, drawing a letter (suchas ‘W’) may cause an application to start (such as a web browser). Thelist of predefined gestures may be preprogrammed by the vendor of thedevice, or trained by the user, or both.

In another embodiment, the trajectory to be matched may be anauthentication signal used to unlock a device, unlock a program runningon a device, or conclude a purchase or other financial transaction. Suchan authentication signal is analogous to a signature, and may be longerthan a simple character or number. Such an air signature may be anabstract set of shapes and curves, or may be a real signature drawn inthe air. The trajectory drawn by the user may be stored in a databasesuch that it can be viewed later if any conflict arises. It may also bematched to the user's predetermined air signature in real time, and theevent intended to be triggered is successfully triggered only if thematch probability is high enough.

To record an arbitrarily long trajectory as an air signature, it may notbe sufficient to use a hard-coded number of features for all users'signatures, as some users may want to use a longer signature thanothers. In this case, the number of features to be used may bedetermined by the number of changes of direction within the signature.

In another embodiment, a hardcoded number of features can be used forall recognizable trajectories, with longer trajectories being made up ofcombinations of shorter trajectories. In this case, a long signaturemust be broken down into shorter units that are recorded and compared,with a long signature being recognized successfully only if it comprisesa set of shorter trajectories that are successfully recognized in thecorrect order.

In one embodiment, the device moved by the user incorporates both themotion sensors and the display on which the feedback trajectory andresult of the gesture recognition may be drawn. In this case, the devicemay be a handset, a portable gaming system, or other electronic systemcomprising a set of motion sensors and a display. In another embodiment,the device may contain motion sensors but not the display, and a displaythat is physically detached from the device may indicate the feedbackand results. For example, the device may be similar to a remote control,air mouse, or game controller, and the display may be a computer, TV, orgame console monitor. FIG. 4 illustrates an exemplary display forproviding the user with a real time feedback. In this example, the userdrew the letter “W” in the air by moving the handheld device 400 and thedisplay 402 displayed the trajectory in real time.

FIG. 5 illustrates an exemplary flow diagram 500 from the training modemethodology for the present invention. In this example, the user hasdrawn the letter “W” in the air while holding the handheld device. AtStep 502, the inertial sensors inside the handheld device provide rawdata related to the “W” trajectory. This data is pre-processed to removenoise etc. and normalized by a processor that can be inside or outsidethe handheld device. At Step 504, the trajectory data is converted intoa set of values corresponding to a set of pre-determined discretefeatures. The types of discrete features included in the features setcan be programmable and can be changed. As an example, one of thediscrete features in the set can include the number of times thetrajectory changes direction. In the case of the letter “W”, the valueof that feature would be three. The values are then stored in adatabase. At Step 506, a command or a character is associated with theset of values representing “W”. In one example, “W” can represent thecommand to open or close a World Wide Web browser. In another example,“W” can simply represent that the character W, which would be typed,entered or displayed in an electronic mail or a word processingdocument. The character or command associated with a set of values canbe user programmable.

FIG. 6 illustrates an exemplary flow diagram 600 for recognizinggesture(s) according to the present invention. In this example, a userdraws the letter “U” in the air while holding a handheld deviceincluding inertial sensors. At Step 602, the inertial sensors providedata related to the “U” trajectory. After pre-processing and filtering,at Step 604, the trajectory data is converted into a set of valuescorresponding to a selected group of discrete features. At Step 606, theset of values is compared with sets of values corresponding to the samegroup of discrete features and a match or a close match is found. As anexample, the rate of change of the trajectory can be a useful featurehere, as that would differentiate between “U” (slower change) and “V”(faster change). At Step 608, a command associated with “U” is executedor the letter “U” is displayed in an application.

FIG. 7 illustrates exemplary computer modules that can be used by thegesture recognition engine 700 of the present invention. The enginecomprises various modules for respectively: obtaining data related totrajectory of movement of a handheld device (702); converting thetrajectory data into a set of values corresponding to selected discretefeatures (704); comparing the converted set of values with sets ofvalues stored in a data store to find a match (706); and displayingcharacter(s) or executing command(s) associated with the match (708).The modules 702-708 can be implemented in hardware, software orfirmware. The modules 702-708 can be implemented on a single computer ormultiple computers. The modules 702-708 can be programmable or hardwired.

In one embodiment, the trajectory drawn may be in the coordinate systemof the handheld device. In this case, a vertically drawn 1 will bevertical relative to the earth if the user is standing up, but may behorizontal if the user is lying down. The trajectory will always be thesame relative to the coordinate system of the handheld device.

In another embodiment, the trajectory drawn may be in the coordinatesystem of the earth or of a separate display not attached to the motionsensors. In this case, a vertically drawn 1 must always be verticalrelative to the earth in order to be correctly recognized as a 1. Inthis case, because the gyroscope data is in the coordinate system of thedevice and not the earth, accelerometer data can be used to determinethe direction of gravity and to process the gyroscope data. This can bedone in real time using sensor fusion techniques or can be done inpost-processing using look-up tables.

In cases in which the user cannot be expected to reliably draw thegesture in the correct coordinate system without some rotational error,the comparison algorithm may be run multiple times using slightlydifferent angular offsets, with the best match chosen as the finalprobability. For example, a trajectory may be compared with a libraryfeatures values sets with rotations of 0, 22.5 degrees, −22.5, 45, and−45.

The user or the device designer may also tune recognition thresholds ina variety of ways. Trajectories may only be valid if they are bigger inspace than a certain threshold, or smaller than a certain threshold.They may only be valid if the execution time of the gesture is longerthan a threshold or shorter than a threshold. They may only be valid ifthe speed of the gesture goes above a threshold or stays below athreshold. Each individual gesture may comprise independent thresholds,and may also comprise a final probabilistic scale factor that weightsthe gesture relative to other gestures, allowing an application designerto pick a gesture from two similar gestures that may be easier totrigger.

According to an aspect of the present invention, in order to recognizelonger gestures, a technique can be used in which long trajectories arebroken down into segments, and each segment is treated like anindividual trajectory. FIG. 8 illustrates an exemplary flow diagram forthe methodology of the present invention for processing long gestures.According to the flow diagram 800, at Step 802, a trajectory made by theuser by waving the handheld device is divided into segments. Forexample, a five letter cursive word can be divided into five segments.The number of segments can include a predetermined number or can bedetermined by analyzing the complexity of the entire trajectory; forexample, calculating the number of changes of directions in the entiretrajectory. At Step 804, a determination is made if each segment isrecognized in the library, i.e. if each segment has a match in thelibrary, within some probability threshold, and a probability valueindicative of the certainty level of the match is assigned to thesegment. If each individual segment has been recognized sufficiently, atStep 806, a final probability value, i.e. the probability valuecalculated for the entire trajectory cumulatively, is calculated. Thefinal probability value can be calculated by combining the individualprobabilities values in various ways; for example, averaging them, ormultiplying them together. At Step 808, a determination is made if thefinal probability value meets or exceeds a final probability threshold.

Determining the final probability threshold that a user must exceed inorder for a gesture to be recognized is difficult because differenttrajectories may require different thresholds, and different users mayhave different degrees of precision in their movement. The probabilitythreshold determination can be automated if a gesture is trained severaltimes. For example, if the gesture ‘M’ is trained five times, thelibrary contains in its memory five iterations of the gesture ‘M’,called M1, M2, M3, M4, and M5. Since the user considers all five ofthese iterations to be the same gesture, it should be possible for thetrajectory M1 to be recognized as M2, M3, M3 and/or M5 within thelibrary. By checking the match probability of M1 versus M2, M1 versusM3, M1 versus M4, and so on, a set of match probabilities can bedetermined. The recommended probability at which a new ‘M’ should berecognized can be determined by combining all the individual matchprobabilities in some way; for example, in an average, or by choosingthe minimum probability.

According to an aspect of the present invention, a user may want totrain a gesture that includes multiple strokes with spaces in between;for example, when training a Chinese, Korean, or Japanese character as agesture. In this case, a mechanism must be in place for determiningwhich movements correspond to “strokes.” For example, the handhelddevice can contain a button that, when held down, signifies that theuser is “drawing” a stroke. When the button is not held down, thissignifies that the user is traveling between strokes. A mechanism canalso be in place that allows the user to see a cursor moving when theuser's hand gesture is traveling between strokes, so that the requiredlocation of the start of the next stroke would be obvious. The user cantherefore draw on a display a multi-stroke character using hand movementin the air.

During training, the each stroke can be stored as a gesture. Thetraveling movements can be stored as <x, y> vectors, signifying astraight line between the end of the previous gesture and the start ofthe new gesture. Each stroke can also have a vector signifying thestraight line between the start of the stroke and the end of the stroke.During operation, all strokes and traveling vectors must match in orderfor the multi-stroke character to match. While the user is deliveringstrokes, an auto-complete feature can be active that attempts to matchthe first few strokes to a database of characters, and determines themost likely matches. When the number of auto-complete suggestions issufficiently small, the user can simply select the correct characterfrom the list. During operation, while a multi-stroke character can havea correct order and direction of strokes, the user may not necessarilydraw the strokes in the right order or in the right direction relativeto the manner in which the multi-stroke character was stored in thelibrary during training. However, the gesture recognition engine of thepresent invention is able to determine that the resulting characterstill matches the desired character.

FIG. 9 illustrates an exemplary technique of the present invention forrecognizing gestures including strokes. The training sequence 910includes a sequence for a three stroke character with strokes A, B, andC. The training sequence 910 is also referred to as the correct strokeorder. The correct stroke order 910 includes vectors vA (from thebeginning of A to the end of A), vAB (travelling between strokes A andB), vB, vBC, and vC. All three strokes and five vectors would be matchedusing the desired machine learning technique, which could be HMM, neuralnetwork, support vector machine, or other similar technique.

During operation, the same character including the strokes A, B and Ccan be drawn in a different, or wrong, order. If a character is drawnwith the strokes in the wrong order 920, for example, B, A, C, thischaracter can also be matched against a transformed version of thestored character in the library. By comparing the character with thestrokes B, A, and C, and the vectors vB, vBA, vA, vAC, and vC, a matchcan be acquired. The new vectors can be calculated by using simplevector math: vAC=vAB+vB+vBC, and vBA=−vB −vAB −vA. In addition, if acharacter is drawn with one or more strokes drawn in the wrongdirection, a match can be made by reversing the individual strokes andtesting the result against the stored library. In this manner, a searchmay be done in which every combination of stroke order and direction canbe tested until a match is found in the library or no match is found.

According to an aspect of the present invention, inertial sensorsincluding gyroscopes are used to provide enhanced power management for ahandheld device. Power management refers to turning off power orswitching a system, a component, a function or an application to alow-power state when inactive. Power management for handheld devices isdesired to increase battery life, reduce cooling requirements, reducenoise, and reduce operating costs for energy and cooling. Lower powerconsumption also means lower heat dissipation, which increases systemstability, decreases energy use, decreases costs and reduces the impacton the environment.

FIG. 10 illustrates a system according to an exemplary embodiment of thepresent invention, in which the inertial sensors are used for powermanagement. The handheld device 1000 includes the main microprocessor1008 coupled to the motion processing unit 1012. The motion processorunit 1012 includes the processor 1014 coupled to the inertial sensormodule 1016, which can include a gyroscope, an accelerometer or both.The inertial sensor module 1016 can also include a compass. Theprocessor 1014 controls the gyro and the accelerometer 1016 andprocesses the information (or data) provided by the gyro and theaccelerometer 1016. The processor 1014 can also be coupled to a compassfor controlling the compass and processing information provided by thecompass. The processor 1014 is also coupled to the programmable module1018 and the main microprocessor 1008. The programmable module 1018 canalso be included as a software module in the main microprocessor 1008.The programmable module can also be included in the motion processingunit 1012.

The main microprocessor 1008 is primarily responsible for managing theoperations of the components of the handheld device 1000. Themicroprocessor 1008 is coupled to the applications (or components)including the network browser 1002 (e.g., an Internet browser), thedisplay 1004, the camera 1006, the microphone 1010 and the keypad 1020.The main microprocessor 1008 is also coupled to the programmable module1018. The programmable module 1008 is coupled to the keypad 1020 and thenetwork 1022 by way of microprocessor 1008. The keypad 1020 can includethe keypad for a smart phone from which the user can program theprogramming module 1018. The network 1022 can include a wired, wirelessor optical network, the Internet, a local area network (LAN) or acomputer from which the programming module 1018 can be programmed.

The display screen 1004 is an essential user interface component and isessential for operating smart phones. The display 1004 is also one ofthe components of the handheld device 1000 that consumes the mostbattery power. According to an aspect of the present invention, themotion processing unit 1012 is used to alleviate the handset's 1000dependency on the display 1004. As illustrated in FIG. 10, the gyroprocessor 1014 and the gyro 1016 are placed in a dedicated powerpartition, separated from the display 1004 and the main processor 1008.The electrical power consumption of the inertial sensors 1016 is muchless than the electrical power consumption of the display 1006 in termsof order of magnitude.

FIG. 11 illustrates an exemplary power management methodology of thepresent invention. According to the flow diagram 1100, at Step 1102, theprogrammable module 1018 is programmed to recognize various “wake up”commands. “Wake up” refers to switching a component of the handhelddevice 1000 from a low-power (i.e. turned off) mode to a high-power(i.e. turned on) mode of operation. The programmable module 1018 can beprogrammed by the user by using the keyboard 1020. The programmablemodule 1022 can also be programmed remotely over the network 1022. Forexample, the “wake up” commands can be uploaded to the programmablemodule 1022 by way of the Internet.

The programmable module can include a look-up table including varioushand motions and their corresponding commands. Examples of the motioncommands can include a shake action or a special moving pattern (i.e. asignature) representing the command to answer the phone, a handwritingmotion of the letter “C” representing the command to turn on the camera,a hand gesture to unlock the phone, a hand gesture to speed dial a phonenumber, and hand motions representing other user-specified shortcuts.For certain motion commands, for example, answering the phone and speeddialing a phone number, the display 1004 and the touch panel 1020 canremain turned off.

At Step 1104, the handheld device 1000 is in the stand-by (or sleep,low-power or turned-off mode). The device 1000 can enter into the sleepmode because, for example, the device 1000 has been idle for more than apre-determined period of time. The sleep mode refers to a low power modein which the handheld device 1000 significantly reduces its powerconsumption, but allows the device 1000 to immediately resume operation(i.e. switch back to active mode) without requiring the user to resetthe programming codes or waiting for the handheld device 1000 to reboot.In the sleep mode, the network browser 1002, display 1004, the camera1006 and the microphone 1010 are turned off and the main microprocessor1008 is throttled down to its lowest-power state. However, according toan aspect of the present invention, the motion processing unit 1012remains running (i.e. active) when the handheld device 1000 is in thesleep mode and continues to monitor for any motion commands.

At Step 1106, the user causes a movement of the handheld device 1000representative of the user's desired command. The movement trajectory isdetected by the gyro 1016. In one embodiment of the present invention,the gyro 1016 generates data representative of the motion trajectory andprovides it to gyro processor 1014. The gyro processor 1014 communicateswith the programmable module 1018 to identify the command or commandscorresponding to the motion trajectory data. The gyro processor 1014provides the main microprocessor 1008 with a wake up signal (Step 1108)and the identification of the user's motion command or commands. Inresponse to the wake up signal, the main microprocessor 1008 switchesfrom the sleep mode to the active (or high-power) mode of operation.

In another embodiment of the present invention, the gyro processor 1014provides the main processor 1008 with the wake up signal and the datarelated to the user's motion's trajectory. The wake up signal causes themain microprocessor 1008 to switch from the sleep mode to the activemode (Step 1108). At Step 1110, the main microprocessor 1008communicates with the programmable module 1018 to identify the commandor commands corresponding to the gyro data. At Step 1112, the mainmicroprocessor 1008 wakes up the application or applications (i.e.causes them to switch from the sleep mode to the active mode) that arerequired to execute the command or commands. At Step 1114, the mainmicroprocessor 1008 and the now-active application or applicationsexecute the command or commands. The present invention thus uses amotion trigger, instead of a push (or soft) button on the handhelddevice, to wake up the handheld device.

In one embodiment of the present invention, only the gyro 1016 is usedto detect the user's motion. Gyros detect angular movements. Becausehuman hand movement is mostly angular and because even linear handmovements are generally accompanied by angular movements, the gyro 1016is well suited for detecting hand motions. As illustrated in FIG. 12,the gyro 1016 can detect any rotational or angular motion around the X,Y and Z axis of the handheld device 1000. Thus, the present inventionprovides the user with a 360 degree freedom along all three axes (X, Yand Z) to program, detect and execute motion commands for the handhelddevice 1000. In another embodiment of the present invention, the user'smotion trajectory is detected by both the gyro 1016 and an accelerometerincluded in the motion processing unit 1012. The gyro 1016 is used fordetecting angular movements associated with the trajectory and theaccelerometer is used for detecting the linear movements associated withthe trajectory. In this embodiment, the processor 1014 can process boththe gyro and accelerometer data.

According to an aspect of the present invention, the motion processingunit 1012 is used as a user interface for detecting a user handmovement. In an exemplary embodiment of the present invention, themotion processing unit 1012 is a sub system of the main microprocessor1008. In another embodiment of the present invention, the motionprocessing unit 1012 is directly controlled by the main microprocessor1008. In both embodiments, the main microprocessor 1008 and the motionprocessing unit 1012 remain in the active mode even when theapplications 1002, 1004, 1006, 1010, 1018 and 1020 enter into the sleepmode, for example, because the handheld device 1000 has been idle forlonger than a predetermined amount of time.

Thus, unlike the embodiment disclosed in FIG. 11, in these embodiments,the main microprocessor 1008 does not need to be awaken-ed as it remainsin the active mode. In response to detecting a movement of the handhelddevice 1000, the main microprocessor 1008-MPU 1012 system automaticallyperforms at least two or all of the following functions: authenticatethe user, identify the command associated with the movement, wake up(i.e. switch from sleep to active mode) the relevant application(s) andexecute the command(s) by using the now awakened application(s).

In the conventional art, for a handheld device such as a mobile phone,to cause the handheld device to switch from sleep to active mode, a usermust push a button on the keypad (or touch panel) of the handhelddevice. That causes the high power consuming display of handheld deviceto switch to active mode, even if the functionality the user desires toperform does not require the use of the display. For example, if theuser is in a meeting and desires to audio tape a portion of the meetingusing his/her smart mobile phone, in the conventional art, the user mustfirst awaken the display and then select the audio taping applicationfor example, by pressing soft or hard key(s) for selecting the audiotaping application, even though the display is not needed for audiotaping. Thus, conventional art causes wastage of power and time andrequires multiple actions by the user.

According to the embodiments of the present invention, the user onlyneeds to make a single predetermined gesture, for example, an airtrajectory in the form of the letter “R” to convey his/her desire tomake an audio recording of the meeting. In response to the “R” motion ofthe handheld device 1000, the main microprocessor 1008-MPU 1012 systemwould automatically communicate with the programmable module 1018 todetermine that the user desires to activate the audio tapingapplication, make a function call to the audio taping application towake up the audio taping application components including, for example,the microphone, and begin the audio recording of the meeting. All of theabove steps are performed automatically in response to the single “R”gesture by the user, and are performed without awakening irrelevantapplications such as the display 1004. The present invention providesadvantages over conventional art in the form of reduced number of steps,less power consumption and reduced time to perform the desired function.

FIG. 13 illustrates an exemplary methodology of the present invention inwhich the camera application is switched from the sleep mode to theactive mode. According to the flow diagram 1300, at Step 1302, thehandheld device 1000 is in the sleep mode and the user sees an objectthat the user would like to photographer. At Step 1304, the user makes a“C” hand gesture in the air while holding the handheld device 1000. Theuser can make the “C” gesture by rotating his/her wrist. The gyro 1016detects the “C” gesture and provides the trajectory data to the gyroprocessor 1014. In one embodiment, the gyro processor 1014 wakes up themain microprocessor 1008. In another embodiment, the main microprocessor1008 is configured to remain in the active mode when the device 1000 isin the sleep mode and directly control the gyro processor 1014. The mainmicroprocessor 1008 communicates with the programmable module 1018 andidentifies the command corresponding to the “C” gesture to include “turnon the camera.”

At Step 1306, the main microprocessor 1008 causes the camera application1006 to switch from sleep to active mode. The main processor 1008 canalso cause the display 1004 to switch from sleep to active mode,depending on the motion commands included in the programming module1018. The network browser 1002 and the microphone 1010 remain in thesleep mode. If the display 1004 is not turned on, the user can view theobject by using the view finder of the camera and use the click buttonto capture the photograph. If both the camera 1006 and the display 1004are turned on, the user can then view the object on the display 1004 andtake the photograph by using the click button. The present inventionprovides the capability to wake up devices and applications inside ahandheld device without requiring push button(s), and specifically byusing the movement of the handheld device as a trigger to wake up thedevices and applications.

FIG. 14 illustrates an exemplary methodology of the present invention inwhich the handheld device is switched from the active mode to the sleepmode and back to the active mode. According to the flow diagram 1400, atStep 1402, the main microprocessor 1008 and the gyro processor 1014 ofthe smart phone 1000 are in the active mode. At Step 1404, the userplaces the phone 1000 face down on a table top, i.e., the display screen1004 and the keypad 1020 of the phone 1000 are face down. At Step 1406,the motion processing unit 1012 detects that the orientation of thephone 1000 is face down. At Step 1408, based on the face downorientation of the phone 1000, the processor 1014, the mainmicroprocessor 1008 and the programmable module 1018 cause themicrophone (or speaker) 1010 to be turned off and the phone to be set inthe mute mode.

At Step 1410, the user causes a movement of the phone 1000, for example,by tapping anywhere on the phone twice to indicate that the user wantsanswer an incoming phone call. The small angular movement of the phone1000 that would result from the tapping is detected by the gyro 1016. Inresponse, at Step 1412, the gyro 1016, the gyro processor 1014, the mainmicroprocessor 1008 and the programmable module 1018 cause the phone1000 to return to the non-mute active mode and turn on the microphone1010.

According to an aspect of the present invention, inertial sensors areused in combination with other sensors to perform user authentication.FIG. 15 illustrates an exemplary handheld device of the presentinvention including inertial and non-inertial sensors. The handhelddevice 1500 includes three types of inertial sensors: gyroscopes (1502),accelerometers (1504) and a compass (1506). In other embodiments, thehandheld device can include either only gyroscopes (1502) or gyroscopes(1502) and accelerometers (1504). The handheld device 1500 also includesother sensors including a camera 1510, RF sensors 1512 and a microphone1514. In other embodiments, the handheld device 1500 can include onlyone or two of these non-inertial sensors 1510, 1512 or 1514. Theprocessing module 1508 can include a microprocessor and memory and isused to process information provided by the sensors 1502, 1504, 1506,1510, 1512 and 1514.

In an embodiment of the present invention, inertial sensors includingone or more gyroscopes, accelerometers and/or compasses, and memory andmicroprocessor for controlling the inertial sensors are located on asubstrate. The substrate can also include an interrupt module to triggerthe applications processor for the handheld device in response to motiondetection by the inertial sensors.

FIG. 16 illustrates an exemplary methodology for combining the dataprovided by the gyroscopes with the data provided by other inertialsensors. The flow diagram 1600 includes the sub-flow diagrams 1610, 1620and 1630. According to the flow diagram 1610, the trajectory informationfor the handheld device 1500 can be obtained from the gyroscopes 1502 inthe reference frame of the handheld device 1500 including the gyroscopes1502 (Step 1612). In this case, it can be sufficient to integrate thepitch and yaw gyroscopes 1502 independently without the use of othersensors, although the other sensors can still be used to calibrate thegyroscopes 1502 (Step 1614).

According to the flow diagram 1620, the trajectory information can beobtained from the gyroscopes 1502 in the reference frame of the Earth(Step 1622). In this case, accelerometers 1504 can be used in additionto gyroscopes 1502 in order to determine the tilt of the handheld device1500 relative to the Earth (Step 1624), and form a trajectory in whichyaw is horizontal relative to the Earth and pitch is vertical relativeto the Earth (Step 1626).

According to the flow diagram 1630, the trajectory information can beobtained from the gyroscopes 1502 in the reference frame of the handhelddevice 1500, without the use of accelerometers 1504 for tiltcompensation (Step 1632). However, a third gyro axis can be used todetect rolling wrist motions (1634), and compensate the trajectory forany distortions due to this rolling motion during the trajectory (Step1636). A magnetometer 1506 can also be used to measure the Earth'smagnetic field, providing a fixed measurement of orientation (Step 1638)that can be used to compensate for any gyroscope 1502 drift (Step 1640).

FIG. 17 illustrates an exemplary methodology of the present invention inwhich angular movement is correlated with linear movement. According tothe flow diagram 1700, at Step 1702, a user causes a movement of thehandheld device 1500. While human body movement is primarily angular,and thus well suited to gyroscope based motion tracking, users mayperform trajectories that include a lot of linear motion due to theirhabits or experiences with linear writing such as writing on ablackboard or writing on a piece of paper with a pencil, both of whichare primarily linear. Because human bodies are joint based, linearmovement will almost always be accompanied by angular movement.

At Step 1704, gyro(s) 1502 are used to obtain information related to theangular movement of the handheld device 1500. At Step 1706,accelerometer(s) 1504 are used to obtain information related to thelinear movement of the handheld device 1500. After sensor fusion betweenaccelerometers 1504 and gyroscopes 1502, linear acceleration can beextracted and double integrated to determine positional changes.However, due to inaccuracies in the sensors, the double integration canbe unstable and may not lead to accurate information about linearmovement. The angular movement, even if it is small, can be used todetermine the movement direction and quality, due to the fact thatangular movement does not require double integration. By correlatingangular movement with linear movement, a final trajectory can becalculated that is usable even if the linear movement that occurs duringthe angular movement is much smaller (Step 1708).

Returning to FIG. 15, according to an aspect of the present invention,the camera 1510 can be used as a motion sensor in order to augment thetrajectory formation. The camera 1510 based motion tracking can useexisting techniques such as cross-correlation, feature tracking, opticalflow, and face tracking. These techniques can provide pixel movementinformation that can be used for trajectory formation. While camera 1510based tracking is usually not reliable by itself, it can be used incombination with accelerometers 1504 and gyroscopes 1502 in order toprovide stable motion tracking. By using gyroscopes 1502 to extract theamount of pixel movement that correlates with angular movement, theremaining pixel movement can be identified as correlating with linearmovement.

A sensor fusion algorithm can be used that uses accelerometer 1504 dataover short linear distances and camera 1510 pixel tracking over longlinear distances in order to provide a more accurate linear trajectory.The camera 1510 tracking can help stabilize the rotational movement byreducing drift. The gyroscopes 1502 and accelerometers 1504 provide moreaccurate movement information over short periods of time, and helpreject disturbances from lighting changes or moving objects within thecamera's 1510 field of view.

According to an aspect of the present invention, RF (radio frequency)sensors 1512 within the handheld device 1500 may also be used for motiontracking by tracking the magnitude of the RF signal and, in some cases,the direction of the RF signal, relative to some external reference1516, such as a RF network, a stationary computer system, or anothersimilar device. The RF signal can include a Bluetooth, Wi-Fi, GPS,infrared, or some other RF signal. The signal tracking can be used tocompliment the other motion sensors by providing a fixed reference tosome outside reference frame. The RF signal can be sent out by thehandheld device 1500, or received by the handheld device 1500, or both.

In one embodiment, the same infrared RF signal can be used to complimentthe motion sensors and also serve as a proximity sensor used todetermine, for example, if a user's face is near the touch screen. Inanother embodiment, a separate infrared signal can be used for eachpurpose. The RF signal can also be used for other transmission purposessuch as controlling a TV, computer, media center, or home automationsystem. The RF signal can be picked up by a dedicated antenna, or by amagnetometer that also serves to measure the Earth's magnetic field.

According to an aspect of the present invention, information from themotion tracking sensors is combined with information from other sensorsthat determine context. The context is used to determine if a commandshould be executed, the nature of the result of the command, or the setof commands that are available. Such context measuring sensors caninclude location sensors such as a global positioning system (GPS), RFsensors that determine the location of the device within an RF network,a touch screen, temperature sensors, a proximity sensor, or amicrophone.

FIG. 18 illustrates an exemplary methodology of the present invention inwhich a microphone and inertial sensors are used for userauthentication. According to the flow diagram 1800, at Step 1802, a usermakes an air signature using the handheld device 1500 including themicrophone 1514. At Step 1804, one or more inertial sensors 1502, 1504and 1506 are used to analyze the user's motion. At Step 1806, adetermination is made if a certain sound or spoken word was detected bythe microphone 1514 at a pre-determined or pre-programmed time includinga time before, during, or after the motion gesture.

The sound can include a sound caused via contact with the device, forexample, by either scratching or tapping the handheld device 1500. Thescratching motion would be very difficult to see and imitate, making theauthentication procedure more secure. The tapping or scratching can beused as a command or as a part of the authentication. The tapping orscratching can be detected by a combination of the information providedby the accelerometer(s) 1504, gyroscope(s) 1502, magnetometer(s) 1506and the microphone (1514). A motion signal detected by a combination ofaccelerometers 1504, gyroscopes 1502, and magnetometers 1506, inresponse to the tapping or scratching, can be combined with an audiosignal detected by the microphone 1514. This can help reject unconscioustapping, for example, unconscious tapping performed by switching hands.This can also make it possible to distinguish between tapping with thepad of the finger and tapping with the nail. At Step 1808, if the user'ssignature matches a signature in a library and the sound detected by themicrophone matches a pre-determined or pre-programmed sound, the user isdeemed to have been successfully authenticated. Thus, both motionsensors and the microphones are used for user authentication.

The motion trajectory that can be captured by the motion processingdevices and sensors illustrated in FIGS. 10 and 15 is a biometric traitand can be used to authenticate a person's identity. As illustrated bymethodology of FIG. 19, an application of motion-based identityauthentication for handset 1000, 1500 is screen unlock. The standarddesign of screen unlock on the conventional smart phones allows a userto input a four digit or letter password or a user-defined line patternto unlock the screen. A drawback of such a touch-panel based screen lockis that an imposter can guess the password based on the fingerprintsmudge of the user. Using motion command to unlock the phone providesthe user with a higher level of security.

According to the flow diagram 1900, at Step 1902, the screen (e.g. 1004in FIG. 10) of the handset 1000, 1500 is locked. At Step 1904, the userperforms an air signature by using the handset 1000, 1500. At Step 1906,the handset 1000, 1500 verifies that the user signature is authentic. AtStep 1908, the handset 1000, 1500 unlocks the screen. Motion-basedidentity authentication has wide range of applications. It can easilyreplace the cumbersome password typing on a smart phone during on-linepurchasing. At a kiosk or a supermarket check-out line, the handset cansend the captured air signature to the cash register via Bluetooth orother wireless communication means.

Moreover, a smart phone is not the only platform that can realizemotion-based identity authentication. The motion processing unitincluding inertial sensors and processor (e.g. 1012 in FIG. 10) isphysically very small and consumes low power such that it can be placedin wearable accessories such as a watch, a keychain, a badge, or a pen.A badge with a motion processing unit transmits ID (identification) aswell as signature pattern. It prevents the risk of someone stealing thebadge to make unauthorized entry. The biometric characteristics providean extra layer of security protection at access control points. Thebiometric authentication method can be applied to access control forhomes, offices and cars, for parental control of TV programs, or forentertainment purposes, for example, in a toy that recognizes its owner.

According to an aspect of the present invention, a motion command doesnot always have to associate with a numerical or alphabetical symbol.The motion command can include any pre-defined motion pattern. Forexample, a video game console can define a special sword-waving sequenceto activate a powerful attack, or determine if the conductor is wavingthe 3-beat or 4-beat pattern in an orchestra game. The motion commandcan be used for both entertainment and educational purposes. Forexample, A golf student can hold the motion processing device andpractice swing to check if the hand grip is turned in the right way.Also, by using a television (TV) remote, a user can write a “C” in theair to change the channel to CNN, or an “E” to change it to ESPN.

Exemplary Networked and Distributed Environments

One of ordinary skill in the art can appreciate that the variousembodiments of methods and devices for pointing based services andrelated embodiments described herein can be implemented in connectionwith any computer or other client or server device, which can bedeployed as part of a computer network or in a distributed computingenvironment, and can be connected to any kind of data store. In thisregard, the various embodiments described herein can be implemented inany computer system or environment having any number of memory orstorage units, and any number of applications and processes occurringacross any number of storage units. This includes, but is not limitedto, an environment with server computers and client computers deployedin a network environment or a distributed computing environment, havingremote or local storage.

FIG. 2 provides a non-limiting schematic diagram of an exemplarynetworked or distributed computing environment. The distributedcomputing environment comprises computing objects 210, 212, etc. andcomputing objects or devices 220, 222, 224, 226, 228, etc., which mayinclude programs, methods, data stores, programmable logic, etc., asrepresented by applications 230, 232, 234, 236, 238. It can beappreciated that objects 210, 212, etc. and computing objects or devices220, 222, 224, 226, 228, etc. may comprise different devices, such asremote controllers, PDAs, audio/video devices, mobile phones, MP3players, laptops, etc.

Each object 210, 212, etc. and computing objects or devices 220, 222,224, 226, 228, etc. can communicate with one or more other objects 210,212, etc. and computing objects or devices 220, 222, 224, 226, 228, etc.by way of the communications network 240, either directly or indirectly.Even though illustrated as a single element in FIG. 2, network 240 maycomprise other computing objects and computing devices that provideservices to the system of FIG. 2, and/or may represent multipleinterconnected networks, which are not shown. Each object 210, 212, etc.or 220, 222, 224, 226, 228, etc. can also contain an application, suchas applications 230, 232, 234, 236, 238, that might make use of an API,or other object, software, firmware and/or hardware, suitable forcommunication with or implementation of the delayed interaction model asprovided in accordance with various embodiments.

There are a variety of systems, components, and network configurationsthat support distributed computing environments. For example, computingsystems can be connected together by wired or wireless systems, by localnetworks or widely distributed networks. Currently, many networks arecoupled to the Internet, which provides an infrastructure for widelydistributed computing and encompasses many different networks, thoughany network infrastructure can be used for exemplary communications madeincident to the techniques as described in various embodiments.

Thus, a host of network topologies and network infrastructures, such asclient/server, peer-to-peer, or hybrid architectures, can be utilized.In a client/server architecture, particularly a networked system, aclient is usually a computer that accesses shared network resourcesprovided by another computer, e.g., a server. In the illustration ofFIG. 2, as a non-limiting example, computers 220, 222, 224, 226, 228,etc. can be thought of as clients and computers 210, 212, etc. can bethought of as servers where servers 210, 212, etc. provide dataservices, such as receiving data from client computers 220, 222, 224,226, 228, etc., storing of data, processing of data, transmitting datato client computers 220, 222, 224, 226, 228, etc., although any computercan be considered a client, a server, or both, depending on thecircumstances. Any of these computing devices may be processing data, orrequesting services or tasks that may implicate the delayed interactionmodel and related techniques as described herein for one or moreembodiments.

A server is typically a remote computer system accessible over a remoteor local network, such as the Internet or wireless networkinfrastructures. The client process may be active in a first computersystem, and the server process may be active in a second computersystem, communicating with one another over a communications medium,thus providing distributed functionality and allowing multiple clientsto take advantage of the information-gathering capabilities of theserver. Any software objects utilized pursuant to the direction basedservices can be provided standalone, or distributed across multiplecomputing devices or objects.

In a network environment in which the communications network/bus 240 isthe Internet, for example, the servers 210, 212, etc. can be Web serverswith which the clients 220, 222, 224, 226, 228, etc. communicate via anyof a number of known protocols, such as the hypertext transfer protocol(HTTP). Servers 210, 212, etc. may also serve as clients 220, 222, 224,226, 228, etc., as may be characteristic of a distributed computingenvironment.

Exemplary Computing Device

As mentioned, various embodiments described herein apply to any devicewherein it may be desirable to perform pointing based services, anddelay interactions with points of interest. It should be understood,therefore, that handheld, portable and other computing devices andcomputing objects of all kinds are contemplated for use in connectionwith the various embodiments described herein, i.e., anywhere that adevice may request pointing based services. Accordingly, the generalpurpose remote computer described below in FIG. 3 is but one example,and the embodiments of the subject disclosure may be implemented withany client having network/bus interoperability and interaction.

Although not required, any of the embodiments can partly be implementedvia an operating system, for use by a developer of services for a deviceor object, and/or included within application software that operates inconnection with the operable component(s). Software may be described inthe general context of computer executable instructions, such as programmodules, being executed by one or more computers, such as clientworkstations, servers or other devices. Those skilled in the art willappreciate that network interactions may be practiced with a variety ofcomputer system configurations and protocols.

FIG. 3 thus illustrates an example of a suitable computing systemenvironment 300 in which one or more of the embodiments may beimplemented, although as made clear above, the computing systemenvironment 300 is only one example of a suitable computing environmentand is not intended to suggest any limitation as to the scope of use orfunctionality of any of the embodiments. Neither should the computingenvironment 300 be interpreted as having any dependency or requirementrelating to any one or combination of components illustrated in theexemplary operating environment 300.

With reference to FIG. 3, an exemplary remote device for implementingone or more embodiments herein can include a general purpose computingdevice in the form of a handheld computer 310. Components of handheldcomputer 310 may include, but are not limited to, a processing unit 320,a system memory 330, and a system bus 321 that couples various systemcomponents including the system memory to the processing unit 320.

Computer 310 typically includes a variety of computer readable media andcan be any available media that can be accessed by computer 310. Thesystem memory 330 may include computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) and/orrandom access memory (RAM). By way of example, and not limitation,memory 330 may also include an operating system, application programs,other program modules, and program data.

A user may enter commands and information into the computer 310 throughinput devices 340. A monitor or other type of display device is alsoconnected to the system bus 321 via an interface, such as outputinterface 350. In addition to a monitor, computers may also includeother peripheral output devices such as speakers and a printer, whichmay be connected through output interface 350.

The computer 310 may operate in a networked or distributed environmentusing logical connections to one or more other remote computers, such asremote computer 370. The remote computer 370 may be a personal computer,a server, a router, a network PC, a peer device or other common networknode, or any other remote media consumption or transmission device, andmay include any or all of the elements described above relative to thecomputer 310. The logical connections depicted in FIG. 3 include anetwork 371, such local area network (LAN) or a wide area network (WAN),but may also include other networks/buses. Such networking environmentsare commonplace in homes, offices, enterprise-wide computer networks,intranets and the Internet.

As mentioned above, while exemplary embodiments have been described inconnection with various computing devices, networks and advertisingarchitectures, the underlying concepts may be applied to any networksystem and any computing device or system in which it is desirable toderive information about surrounding points of interest.

There are multiple ways of implementing one or more of the embodimentsdescribed herein, e.g., an appropriate API, tool kit, driver code,operating system, control, standalone or downloadable software object,etc. which enables applications and services to use the pointing basedservices. Embodiments may be contemplated from the standpoint of an API(or other software object), as well as from a software or hardwareobject that provides pointing platform services in accordance with oneor more of the described embodiments. Various implementations andembodiments described herein may have aspects that are wholly inhardware, partly in hardware and partly in software, as well as insoftware.

The word “exemplary” is used herein to mean serving as an example,instance, or illustration. For the avoidance of doubt, the subjectmatter disclosed herein is not limited by such examples. In addition,any aspect or design described herein as “exemplary” is not necessarilyto be construed as preferred or advantageous over other aspects ordesigns, nor is it meant to preclude equivalent exemplary structures andtechniques known to those of ordinary skill in the art. Furthermore, tothe extent that the terms “includes,” “has,” “contains,” and othersimilar words are used in either the detailed description or the claims,for the avoidance of doubt, such terms are intended to be inclusive in amanner similar to the term “comprising” as an open transition wordwithout precluding any additional or other elements.

As mentioned, the various techniques described herein may be implementedin connection with hardware or software or, where appropriate, with acombination of both. As used herein, the terms “component,” “system” andthe like are likewise intended to refer to a computer-related entity,either hardware, a combination of hardware and software, software, orsoftware in execution. For example, a component may be, but is notlimited to being, a process running on a processor, a processor, anobject, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running oncomputer and the computer can be a component. One or more components mayreside within a process and/or thread of execution and a component maybe localized on one computer and/or distributed between two or morecomputers.

The aforementioned systems have been described with respect tointeraction between several components. It can be appreciated that suchsystems and components can include those components or specifiedsub-components, some of the specified components or sub-components,and/or additional components, and according to various permutations andcombinations of the foregoing. Sub-components can also be implemented ascomponents communicatively coupled to other components rather thanincluded within parent components (hierarchical). Additionally, itshould be noted that one or more components may be combined into asingle component providing aggregate functionality or divided intoseveral separate sub-components, and any one or more middle layers, suchas a management layer, may be provided to communicatively couple to suchsub-components in order to provide integrated functionality. Anycomponents described herein may also interact with one or more othercomponents not specifically described herein but generally known bythose of skill in the art.

In view of the exemplary systems described supra, methodologies that maybe implemented in accordance with the disclosed subject matter will bebetter appreciated with reference to the flowcharts of the variousfigures. While for purposes of simplicity of explanation, themethodologies are shown and described as a series of blocks, it is to beunderstood and appreciated that the claimed subject matter is notlimited by the order of the blocks, as some blocks may occur indifferent orders and/or concurrently with other blocks from what isdepicted and described herein. Where non-sequential, or branched, flowis illustrated via flowchart, it can be appreciated that various otherbranches, flow paths, and orders of the blocks, may be implemented whichachieve the same or a similar result. Moreover, not all illustratedblocks may be required to implement the methodologies describedhereinafter.

While the various embodiments have been described in connection with thepreferred embodiments of the various figures, it is to be understoodthat other similar embodiments may be used or modifications andadditions may be made to the described embodiment for performing thesame function without deviating therefrom. Still further, one or moreaspects of the above described embodiments may be implemented in oracross a plurality of processing chips or devices, and storage maysimilarly be effected across a plurality of devices. Therefore, thepresent invention should not be limited to any single embodiment, butrather should be construed in breadth and scope in accordance with theappended claims.

What is claimed is:
 1. A hand held electronic device, comprising: agyroscope that generates gyroscope data representing a trajectory of amovement of the handheld device; an accelerometer that generatesaccelerometer data representing a tilt of the handheld device relativeto earth's gravity, wherein the trajectory comprises at least onesegment; and a processing component that generates a command to initiatea function that corresponds to the command in the hand held electronicdevice; wherein generating a command comprises: moving the handhelddevice along a selected trajectory corresponding to the command;determining the selected trajectory and segments thereof using at leastone of the gyroscope and the accelerometer; processing data from the atleast one of the gyroscope and the accelerometer to derive dataparameters; wherein the data parameters include any of at least one ofan angle of the trajectory, an inverse tangent of a change in pitchdivided by a change in yaw, a location of a crossing point or a numberof times in which the trajectory changes in direction; and determiningthe command from the determined selected trajectory and segments thereofbased on the derived data parameters to initiate the function in thehandheld device.
 2. The handheld device of claim 1, wherein theprocessing component stores at least a portion of the data generated bythe at least one of the gyroscope and the accelerometer in a data store.3. The handheld device of claim 1, further comprising a display fordisplaying the trajectory of the movement.
 4. The handheld device ofclaim 1, wherein the processing component executes a command associatedwith an application program associated with the handheld device or anoperating system associated with the handheld device.
 5. The handhelddevice of claim 4, wherein the command triggers an event in a videogame.
 6. The handheld device of claim 4, wherein the command opens orcloses the application program or controls a set of applications runningwithin the operating system.
 7. A method of generating a command toinitiate a function that corresponds to the command in a hand heldelectronic device; the method comprising: moving the handheld devicealong a selected trajectory corresponding to the command; determiningthe selected trajectory and segments thereof using at least one of agyroscope and an accelerometer contained within the handheld device;processing data from the at least one of the gyroscope and theaccelerometer to derive data parameters; wherein the data parametersinclude any of at least one of an angle of the trajectory, an inversetangent of a change in pitch divided by a change in yaw, a location of acrossing point or a number of times in which the trajectory changes indirection; and determining the command from the determined selectedtrajectory and segments thereof based on the derived data parameters toinitiate the function in the handheld device.
 8. The method of claim 7,wherein the processing component stores at least a portion of the datagenerated by the at least one of the gyroscope and the accelerometer ina data store.
 9. The method of claim 7, further comprising displayingthe trajectory of the movement on a display in the hand held electronicdevice.
 10. The method of claim 7, wherein the processing componentexecutes a command associated with an application program associatedwith the handheld device or an operating system associated with thehandheld device.
 11. The method of claim 10, wherein the commandtriggers an event in a video game.
 12. The method of claim 10, whereinthe command opens or closes the application program or controls a set ofapplications running within the operating system.
 13. A hand heldelectronic device, comprising: a gyroscope that generates gyroscope datarepresenting a trajectory of a movement of the handheld device; anaccelerometer that generates accelerometer data representing a tilt ofthe handheld device relative to earth's gravity, wherein the trajectorycomprises at least one segment; a compass that generates compass datarepresenting a direction of the handheld device relative to the earth'smagnetic pole; and a processing component that generates a command toinitiate a function that corresponds to the command in the hand heldelectronic device; wherein generating a command comprises: moving thehandheld device along a selected trajectory corresponding to thecommand; determining the selected trajectory and segments thereof usingat least one of the gyroscope, the accelerometer and the compass;processing data from the at least one of the gyroscope, theaccelerometer and the compass to derive data parameters; wherein thedata parameters include any of at least one of an angle of thetrajectory, an inverse tangent of a change in pitch divided by a changein yaw, a location of a crossing point or a number of times in which thetrajectory changes in direction, and a direction of the handheld devicerelative to the earth's magnetic pole; and determining the command fromthe determined selected trajectory and segments thereof based on thederived data parameters to initiate the function in the handheld device.14. The handheld device of claim 13, wherein the processing componentstores at least a portion of the data generated by the at least one ofthe gyroscope, the accelerometer and the compass in a data store. 15.The handheld device of claim 13, further comprising a display fordisplaying the trajectory of the movement.
 16. The handheld device ofclaim 13, wherein the processing component executes a command associatedwith an application program associated with the handheld device or anoperating system associated with the handheld device.
 17. The handhelddevice of claim 16, wherein the command triggers an event in a videogame.
 18. The handheld device of claim 16, wherein the command opens orcloses the application program or controls a set of applications runningwithin the operating system.